Method of improving the availability of a computer clustering system through the use of a network medium link state function

ABSTRACT

A method for increasing the availability of a first server included in a computer cluster when a second server fails. Each server in the computer cluster has an associated mass storage device and can process requests from any network device in the computer cluster. Data is mirrored between the mass storage devices of the servers so that each server&#39;s mass storage device has a complete copy of all computer cluster data. Data mirroring takes place across a dedicated link, which reduces congestion on the rest of the computer cluster. When the first server detects a loss of communication from the second server, the first server determines if the loss of communication is a result of a malfunction of the dedicated link. If the dedicated link has failed, the first server discontinues operation to avoid writing data to its associated mass storage device, which cannot be mirrored due to the loss of communication. If the dedicated link is operational, the first server continues operation. In either case, since each server can process requests from any network device and each server has a complete copy of all the network data, the computer cluster continues to be available for use even after a server is shut down.

RELATED APPLICATION

[0001] This application claims the benefit of U.S. Provisional PatentApplication Serial No. 60/257,478, entitled “Maintaining Operation of aNetwork Server after Failure of another Network Server,” filed Dec. 21,2000, which is incorporated herein by reference.

BACKGROUND OF THE INVENTION

[0002] 1. The Field of the Invention

[0003] This invention relates to computer clustering systems and inparticular to methods for improving the availability and reliability ofcomputer clustering system resources and data in the event of loss ofcommunication between computer clustering system servers.

[0004] 2. Description of Related Art

[0005] A typical computer cluster includes two or more servers and oneor more network devices in communication with each other across acomputer network. During normal operation of a computer cluster, theservers provide the network devices with computer resources and a placeto store and retrieve data. In current computer cluster configurationsthe computer cluster data is stored on a shared computer disk that isaccessed by any of the network servers.

[0006] A typical computer cluster is illustrated in FIG. 1, whichillustrates two network servers 110 and 120 in communication withnetwork devices 130, 140, and 150 across computer network 101. Bothnetwork server 110 and network server 120 communicate with shared disk104 across communication lines 105 and 106, respectively.

[0007] When using a computer cluster, it is often desirable to providecontinuous availability of computer cluster resources, particularlywhere a computer cluster supports a number of user workstations,personal computers, or other network client devices. It is also oftendesirable to maintain uniform data between different file serversattached to a computer clustering system and maintain continuousavailability of this data to client devices. To achieve reliableavailability of computer cluster resources and data it is necessary forthe computer cluster to be tolerant of software and hardware problems orfaults. Having redundant computers and a mass storage device generallydoes this, such that a backup computer or disk drive is immediatelyavailable to take over in the event of a fault.

[0008] A technique currently used for implementing reliable availabilityof computer cluster resources and data using a shared disk configurationas shown in FIG. 1 involves the concept of quorum, which relates to astate in which one network server controls a specified minimum number ofnetwork devices such that the network server has the right to controlthe availability of computer resources and data in the event of adisruption of service from any other network server. The manner in whicha particular network server obtains quorum can be conveniently describedin terms of each server and other network devices casting “votes”. Forinstance, in the two server computer cluster configuration of FIG. 1,network server 110 and network server 120 each casts one vote todetermine which network server has quorum. If neither network serverobtains a majority of the votes, shared disk 104 then casts a vote suchthat one of the two network servers 110 and 120 obtains a majority, withthe result that quorum is obtained by one of the network servers in amutually understood and acceptable manner. Only one network server hasquorum at any time, which ensures that only one network server willassume control of the entire network if communication between thenetwork servers 110 and 120 is lost.

[0009] The use of quorum to attempt to make network servers available inthe event of a disruption will now be described. There are two generalreasons for which server 110 can detect a loss of communication withserver 120. The first is an event, such as a crash, at server 120, inwhich server 120 is no longer capable of providing network resources toclients. The second is a disruption in the communication infrastructureof network 101 between the two servers, with server 120 continuing to becapable of operating within the network. If server 110 can no longercommunicate with server 120, its initial operation is to determine if ithas quorum. If server 110 determines that it does not have quorum, itthen attempts to get quorum by sending a command to shared disk 104requesting the disk to cast a vote. If shared disk 104 does not vote forserver 110, this server shuts itself down to avoid operatingindependently of server 120. In this case, server 110 assumes thatnetwork server 120 is operating with quorum and server 120 continues tocontrol the computer cluster. However, if shared disk 104 votes fornetwork server 110, this server takes quorum and control of the computercluster and continues operation under the assumption that network server120 has malfunctioned.

[0010] While the use of quorum to enable one of a plurality of networkservers to continue providing network resources in the event of adisruption in the network is often satisfactory, the use of a shareddisk places the entire network and the data stored on the disk at riskof being lost. For instance, if the shared disk 104, rather than one ofthe network servers 110 and 120 malfunctions, neither of the servers canoperate, and the data may be permanently lost. Moreover, in a shareddisk configuration the computer cluster servers are typically placed inclose proximity to each other. This creates the possibility that naturaldisasters or power failures may take down the whole computer cluster.

SUMMARY OF THE INVENTION

[0011] The present invention relates to a method for improving theavailability and reliability of computer cluster resources and data in acomputer clustering system. Two servers each having an associated diskcommunicate across a computer network. Each server is capable ofproviding computer cluster resources and accessing computer cluster datafor all network devices attached to the computer network. In the eventof loss of communication, each server has the ability to determine thereason for loss of communication and determine whether or not it shouldcontinue operation.

[0012] When a network server detects that communication with anothernetwork server is lost, the loss in communication can be due to either afailure of the communication link or a failure of the other networkserver. Because each network server has a mirrored copy of the networkdata, a loss in communication is followed by execution of a series ofacts at each network server that remains operating to ensure that thenetwork servers do not begin operating independently of each other. Inthe absence of these acts, multiple network servers operatingindependently of one another could exist in an undesirable “split brain”mode, in which data mirroring between the network servers is notperformed, thereby resulting in potential data corruption.

[0013] When operation of the computer cluster is initiated, one serveris assigned control of the computer cluster resources and data and isgiven a “right to survive” in the event that communication between thenetwork servers is lost as a result in failure of the communicationlink. For convenience, the one network server that has the “right tosurvive” during normal operation is designated herein as a “primary”server and any server that is not does not have the right to surviveduring normal operation is designated as a “secondary” server. It isnoted that the terms “primary” and “secondary” do not connote relativeimportance of the servers, nor do they refer to which server isprimarily responsible for providing network resources to networkdevices. Under normal operation, primary and secondary servers can beinterchangeable from the standpoint of providing network resources. Theright to survive is used in a default protocol to ensure that the splitbrain problem does not arise if communication between network servers islost.

[0014] When a primary network server detects loss of communication, theprimary network server can continue operating, since it can assume thatthe other, secondary network server has failed or that the secondarynetwork server will not continue operation. The series of acts performedby a secondary network server upon detecting loss of communication issomewhat more complex. Rather than simply ceasing operation, thesecondary network server infers or determines whether the loss ofcommunication is a result of a failure of the primary network server ora failure in the communication link. If the communication link isoperational, the secondary network server concludes that the primarynetwork server has failed and is not operating. In this case, thesecondary network server continues operating substantially without therisk of causing the split brain problem. If, however, the secondarynetwork server determines that communication link has failed, it assumesthat the primary network server is operational. In response to thisassumption, the secondary network server terminates operation to avoidoperating in a split brain mode.

[0015] A significant benefit according to the invention is that asecondary server, which does not initially have right to survive, cancontinue operating if it determines that a loss of communication withthe primary server is not caused by failure of the communication link.In the absence of any analysis of the communication link, the secondaryserver would be required to automatically shut down in response to aloss of communication with the primary server to avoid the split brainproblem. It is noted that the foregoing methods of the invention forresponding to loss of communication between servers enhances thereliability and availability of computer clusters in which each networkserver has a dedicated, mirrored disk or mass storage device, since thepossibility of operating in a split brain mode does not force asecondary server to go off-line when a primary server fails.

[0016] Application of conventional “quorum” rules to computer clustersin which each network server has a dedicated, mirrored disk, isgenerally not optimal. For instance, in the case where a network serverhaving quorum fails, there is no shared disk to cast a vote that wouldreassign quorum to the other network server. As a result, the directapplication of conventional quorum rules to this type of computercluster would result in the non-quorum network server unnecessarilyshutting down upon failure of the network server having quorum.

[0017] Storing data in separate, mirrored disks greatly reduces thepossibility of losing network data, which has been a problem frequentlyexperienced in computer clusters having a single disk that is sharedamong network servers. Additionally, since servers do not share a singledisk according to the invention, the location of the servers is notlimited by the cable lengths associated with disk drive devices. Thus,network servers and their associated mirrored disks can be locatedremotely one from another. This reduces the chance that naturaldisasters or power failures may disable the entire computer cluster.

[0018] Additional features and advantages of the invention will be setforth in the description which follows, and in part will be obvious fromthe description, or may be learned by the practice of the invention. Thefeatures and advantages of the invention may be realized and obtained bymeans of the instruments and combinations particularly pointed out inthe appended claims. These and other features of the present inventionwill become more fully apparent from the following description andappended claims, or may be learned by the practice of the invention asset forth hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order that the manner in which the above-recited and otheradvantages and features of the invention are obtained, a more particulardescription of the invention briefly described above will be rendered byreference to specific embodiments thereof which are illustrated in theappended drawings. Understanding that these drawings depict only typicalembodiments of the invention and are not therefore to be considered tobe limiting of its scope, the invention will be described and explainedwith additional specificity and detail through the use of theaccompanying drawings in which:

[0020]FIG. 1 is a block diagram illustrating a conventional computercluster having servers that share a disk.

[0021]FIG. 2 illustrates an exemplary computer cluster that provides asuitable operating environment for the present invention.

[0022]FIG. 3 illustrates communication between the different softwaremodules in a server to enable the server to decide whether to assumeright to survive.

[0023]FIG. 4 is a flow diagram illustrating a method whereby a serverdetermines whether it is to assume the right to survive upon detectingloss of communication with the other server.

[0024]FIG. 5 is a flow diagram illustrating a method whereby a serverhaving the right to survive responds to a failure in another server in acomputer cluster.

DETAILED DESCRIPTION OF THE INVENTION

[0025] The present invention relates to a method for improving theavailability and reliability of computer cluster resources and data in acomputer clustering system. The computer cluster includes at least twoservers, each having a dedicated mass storage devices. The serverscommunicate with each other as well as other network devices across acomputer network. Data is mirrored between the disks of each server sothat network devices have access to reliable data in the event of oneserver failing. Communication modules operate on each server todetermine whether a server should shut itself down in the event of acommunication loss between the servers.

[0026] The term “right to survive” refers to whether or not a server hasthe right to continue operation in the event of a detected loss ofcommunication between the servers due to an error in the communicationlink between the servers. For example, if loss of communication betweenthe two servers is detected, and such loss is due to a physical break inthe communication link, the server with the right to survive continuesoperation while the server without right to survive shuts itself down.As noted previously, a “primary” server is one that has the right tosurvive prior to loss of communication, whereas a “secondary” server isone that does not have the right to survive prior to the loss ofcommunication between servers.

[0027] The term “split brain” refers to an undesirable condition inwhich the network servers of a computer cluster having dedicated massstorage devices for each network server act independently of each otherand without mirroring. In the absence of the invention, this can occurwhen communication between the network servers is lost while bothnetwork servers are still running, and each node assumes that the otherhas failed. When split brain occurs, the mirrored data on each server nolonger matches and can be corrupt. Referring to the above example wherethe servers lose communication due to a physical break in thecommunication link, if the servers could not determine the loss was duethe physical break, each would continue to operate and write its owndata to its associated mass storage device. However, since thecommunication link is broken, the data would not be mirrored andinconsistent data on the mass storage devices would result. The presentinvention prevents the computer cluster from operating in a split brainmode, while enabling the secondary server to continue operating if theloss of communication is caused by failure of the primary server.

[0028] Embodiments within the scope of the present invention alsoinclude computer-readable media for carrying or having stored thereoncomputer-executable instructions or data structures. Suchcomputer-readable media can be any available media, which can beaccessed by a general purpose or special purpose computer. By way ofexample, and not limitation, such computer-readable media can compriseRAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic diskstorage or other magnetic storage devices, or any other medium which canbe used to carry or store desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer. Wheninformation is transferred or provided over a network or anothercommunications connection (either hardwired, wireless, or a combinationof hardwired or wireless) to a computer, the computer properly views theconnection as a computer-readable medium. Thus, any such a connection isproperly termed a computer-readable medium. Combinations of the aboveshould also be included within the scope of computer-readable media.Computer-executable instructions comprise, for example, instructions anddata which cause a general purpose computer, special purpose computer,or special purpose processing device to perform a certain function orgroup of functions.

[0029]FIG. 2 and the following discussion are intended to provide abrief, general description of a suitable computing environment in whichthe invention may be implemented. Although not required, the inventionwill be described in the general context of computer-executableinstructions, such as program modules, being executed by computers innetwork environments. Generally, program modules include routines,programs, objects, components, data structures, etc. that performparticular tasks or implement particular abstract data types.Computer-executable instructions, associated data structures, andprogram modules represent examples of the program code means forexecuting steps of the methods disclosed herein. The particular sequenceof such executable instructions or associated data structures representexamples of corresponding acts for implementing the functions describedin such steps.

[0030] Those skilled in the art will appreciate that the invention maybe practiced in network computing environments with many types ofcomputer system configurations, including personal computers, hand-helddevices, multi-processor systems, microprocessor-based or programmableconsumer electronics, network PCs, minicomputers, mainframe computers,and the like. The invention may also be practiced in distributedcomputing environments where tasks are performed by local processingdevices and remote processing devices that are linked (either byhardwired links, wireless links, or by a combination of hardwired orwireless links) through a communications network. In a distributedcomputing environment, program modules may be located in both local andremote memory storage devices.

[0031]FIG. 2 illustrates a representative computer cluster configurationin which the method of the invention can be practiced. The computercluster includes two servers, which are designated as server A 210 andserver B 220. Although only two servers are illustrated in FIG. 2, thegeneral principles disclosed herein can be readily adapted to computerclusters having more than two network servers. Server A 210 and server B220 both run a file operating system, which may be Microsoft Windows NT,although any of a variety of operating systems can be used with theinvention. Server A 210 includes computer 212 connected to network 201through network interface 211 and mass storage device 214 connectedthrough mass storage controller 213. Likewise, server B 220 includescomputer 222 connected to network 201 through network interface 220 andmass storage device 224 connected through mass storage controller 223.Network 201 can be an Ethernet, token ring, Arcnet, or any other networkby which server A 210 and server B 220 can communicate with networkdevices 230, 240, and 250.

[0032] While it is not necessary for server A 210 to have identicalcomponents to server B 220, in many instances this will be the case. Inother instances, server A 210 and server B 220 may have differentprocessor types, different processor speeds, different size mass storagedevices or any other number of hardware differences. All that isrequired of server A 210 and server B 220 is that they be capable ofrunning the file operating system and that the drive on one of theservers not be so large that it cannot be mirrored to the other server.

[0033] In the embodiment illustrated in FIG. 2, communication betweenserver A 210 and server B 220 is established using dedicated link 202.Computer 212 is connected with dedicated link 202 through communicationattachment 215 and computer 222 is connected with dedicated link 202through communication attachment 225. Dedicated link 202 can beimplemented using a variety of techniques, well known to those skilledin the art. In one embodiment, dedicated link 212 is a link that uses anEthernet protocol. Alternatives include using the serial communicationsports of computers 212 and 222 programmed to run at high speeds or theparallel interfaces of computers 212 and 222. According to anotheralternative, dedicated link 202 and communication attachments 215 and225 are not present, with the communication between server A 210 andserver B 220 being established by a virtual circuit or channel carriedacross network 201. The specific type of hardware used to implementdedicated link 202 is not important, provided data transfer rates arecomparable to the data transfer rates on mass storage devices 214 and224 so performance of the system is not limited.

[0034] Network devices 230, 240 and 250 connect to network 201 throughnetwork interfaces 231, 241 and 251 respectively. These are clientdevices that use the resources of computer systems 210 and 220 to accessdata stored on the mass storage devices 214 and 224. Network devices230, 240 and 250 can be any devices capable of communicating acrossnetwork 201.

[0035] During operation of the computer cluster, both server A 210 andserver B 220 are capable of providing resources to any of networkdevices 230, 240, or 250. Furthermore, both server A 210 and server B220 are capable of sensing errors in the integrity of dedicated link202. When a network device in communication with server A 210 adds,changes or deletes data from mass storage device 214, the update ismirrored across dedicated link 202 to mass storage device 224. Likewise,when a network device in communication with server B 220 adds, changesor deletes data from mass storage device 224 the update is mirroredacross dedicated link 202 to mass storage device 214. Since the data ismirrored across dedicated link 202 in the embodiment of FIG. 2, it doesnot congest network 201.

[0036] A result of mirroring is that mass storage devices 214 and 224contain identical copies of all the computer cluster data. Therefore,the computer cluster can continue to operate reliably if one of the massstorage devices malfunctions. If network devices 230, 240 and 250 cannotaccess data on mass storage 214 as a result of a malfunction of massstorage device 214, they may access the data on mass storage device 224and vice versa. Since server A 210 and server B 220 do not share a massstorage device, there is no single point of failure and they may beremotely located from each other. Remotely locating server A 210 andserver B 220 reduces the chance of a power outage or natural disasterpreventing access to both mass storage devices simultaneously. Furtherdetails regarding the basic mirroring capabilities and general networkarchitecture of the computer cluster of FIG. 2 are presented in U.S.Pat. No. 5,978,565 entitled “METHOD FOR RAPID RECEOVERY FROM A NETWORKFILE SERVER FAILRURE INCLUDING METHOD FOR OPERATING CO-STANDBY SERVERS,”issued, Nov. 2, 1999, which is incorporated herein by reference.

[0037] When server A 210 and server B 220 are initially configured, oneof the servers is assigned the right to survive by the operator. Theright to survive in this instance is represented as a software flag andis set to either “on” or “off.” In general, during normal operation ofthe computer cluster, the right to survive flag is set to “on” on oneand only one of the servers. After the initial configuration, a serverwith its right to survive flag “off” may turn its right to survive flag“on” under certain network conditions, as will be disclosed in greaterdetail below.

[0038] If communication between the servers is lost, the right tosurvive is used in determining which server is to continue clusteroperations and which server is to terminate cluster operations. Thereare two primary instances when loss of communication between server A210 and server B 220 occurs. The first instance occurs when one of theservers ceases communicating across dedicated link 202 as a result, forexample, of the server experiencing a software error or the serverlosing power. The second instance occurs when there is a failure in theintegrity of dedicated link 202.

[0039] As stated above, server A 210 and server B 220 can distinguishbetween types of communication failure. In the following discussion, itis assumed that server B 220 is a primary server, meaning that it hasright to survive during an initial period of normal operation of thecomputer cluster, while server A 210 is a secondary server that does notinitially have right to survive. If the primary server B 220 detects aloss of communication from server A 210, server B 220 continuesoperating in view of its right to survive without regard to whether theloss of communication has been caused by a failure of communication link202 or of server A 210.

[0040] When secondary server A 210 detects a loss of communication fromserver B 220, server A 210 discontinues operation unless it determinesthat it is safe for it to assume right to survive. If server A 210determines that the loss of communication is due to a failure incommunication link 202, server A 210 assumes that primary server B 220is operational. Accordingly, because server A 210 does not have right tosurvive, it terminates its activity so as to avoid operating in a splitbrain mode. If, however, secondary server A 210 determines thatcommunication link 202 has not failed, it assumes that the loss ofcommunication is associated with primary server 220 B failing orotherwise no longer providing network services. In this case, secondaryserver A 210 assigns itself right to survive and continues operationwithout the risk of causing a split brain in the computer cluster. Inthe latter case, server A 210 can service requests for resources fromnetwork devices 230, 240, and 250 that would otherwise be made to thefailed server B 220 In any of the foregoing situations, only one servercontinues to function after loss of communication is experienced betweenthe servers, which prevents the problem of split brain from occurring.

[0041]FIG. 3 shows the interaction of different software modules runningon a representative computer system according to one embodiment of theinvention. The systems and structures of FIG. 3 represent one example ofhow server A 210 can determine whether it should continue functioningand assume the right to survive upon detecting loss of communicationwith the other server. Right to survive flag 306 is initially set,either automatically or by a system administrator when the operation ofthe computer cluster is initiated. As stated above, the server with itsright to survive flag 306 set to “on” continues functioning in the eventof loss of communication with another server, regardless of whether thecause is an error in the integrity of the dedicated link 202 or afailure of the other server. While the invention broadly disclosedherein extends to a primary server that has its right to survive flag306 initially set in the “on” position and that subsequently continuesoperation after experiencing a loss of communication, FIG. 3 isdiscussed in detail below in the context of a secondary server thatloses communication while the right to survive flag 306 is set in the“off” position.

[0042] As server A 210 operates in the computer cluster, the serverreceives and transmits data between components of the computer clusteras shown in FIG. 3. Data 308 is representative of a heartbeat signal ofserver B 220 communicated on dedicated link 202 to server A 210. As usedherein, the term “heartbeat signal” extends to any signal or data havingany format that can used by one server to determine whether anotherserver is operational. Communication attachment driver 301 is a softwaremodule that controls communication attachment 215 and receives data 308.Likewise, data 309 is representative of data transmitted on network 201.Network interface driver 302 controls network interface 211 and receivesdata 309.

[0043] Server A 210 can use data 308 and the modules that process andanalyze data 308 and dedicated link 210 to determine whether to continueor discontinue operation in response to a determination thatcommunication with server B 220 has been lost at a moment at which rightto survive flag 306 is set in the “off” position. Server A 210 includesa server communication detector that monitors communication with serverB 220 to determine whether data 308 is being received as expected. Linkstate detector 304 is any hardware component, software component, or anycombination thereof that is capable of determining if dedicated link 202has failed when communication with server B 220 is lost. In many cases,server communication detector 303 and link state detector 304 can beseparate modules, although the functionality of these two components canbe combined in a single structure.

[0044] In this example, communication detector 303 and link statedetector 304 monitor data 308 and the integrity of dedicated link 202.However, in the alternative embodiment in which communication withserver B 220 is transmitted using network 201 and network interfacedriver 302 rather than using dedicated link 202, the servercommunication detector 303 and the link state detector 304 monitor data309 and the integrity of network 201.

[0045] In either case, server communication detector 303 determineswhether the heartbeat signal of server B 220 is detected. In the case ofa loss of the heartbeat signal of server B 220, link state detectoranalyzes the integrity of the communication link that had been carryingthe heartbeat signal (e.g., dedicated link 202 or network 201). Whenloss of communication with server B 220 is experienced, servercommunication detector 303 and link state detector 304 notifycommunication manager 305 of this fact and whether it appears that thecommunication link that had been carrying the heartbeat signal hasfailed. Loss of communication manager 305 then determines whether toshut server A 210 down or to permit server A to continue operating basedon the right to survive flag 306 and the information received fromserver communication detector 303 and from link state detector 304.

[0046] To summarize the rules applied by loss of communication manager305, server A 210 is permitted to continue operating if right to surviveflag 306 indicates that server A 210 has the right to survive. Assuming,however, that server A 210 is a secondary server that does not have theright to survive according to right to survive flag 306 at the time thatcommunication with server B 220 was lost, loss of communication manager305 discontinues the operation of server A 210 if it has been determinedthat the loss of communication was a result of failure of thecommunication link. If loss of communication has been caused by failureof the communication link, it is assumed that server B 220 isoperational, in which case, server A 210 discontinues operation, therebyavoiding the possibility of operating the computer cluster in a splitbrain mode.

[0047] In contrast, if loss of communication manager 305 determines thatserver A 210 detects a loss of communication from server B 220 and thecommunication link has not failed, loss of communication manager 305infers that server B 220 has malfunctioning. Under the assumption thatserver A 210 is a secondary server, the right to survive flag 306 isthen set to the “on” position, indicating that server A 210 has assumedthe right to survive. In this case, server A 210 “reforms” the computercluster by beginning to service requests previously made to server B220. Server A 210 can do so by sending commands to network interfacedriver 302, causing network interface driver 302 to service requeststhat network devices have sent to server B 220 on network 201.

[0048]FIG. 4 is a flow diagram showing steps performed at a secondaryserver (i.e., a server that does not already have the right to survive)in a computer cluster according to one embodiment of the invention forincreasing the availability of a network resources in the event of afailure of a primary server (i.e., a server initially having the rightto survive). It is noted that a primary server generally does not needto perform the steps illustrated in FIG. 4 upon losing communicationwith a secondary server, as the right to survive possessed by primaryserver enables it to continue operating regardless of the reason for theloss of communication, as will be described below in greater detail inreference to FIG. 5.

[0049] Turning first to FIG. 4, the secondary server A attempts todetect reliable communication with server B in decision block 401. Ifserver A can detect reliable communication with server B, the methodproceeds to step 402, in which server A waits a certain time intervaland then repeats decision block 401.

[0050] If server A does not detect reliable communication with server Bin decision block 401, server A checks the reliability of thecommunication link in decision block 403. If server A does not detect areliable communication link in decision block 402 server A terminatescluster activity at step 405 so as to avoid operating in a split brainmode..

[0051] If, however, server A does detect a reliable communication linkat decision block step 403, the method proceeds to decision block 407,in which it is determined whether server A has waited the requirednumber of time intervals before it can assume that server B has failed.If, according to decision block 407, server A has not waited therequired number of time intervals, the method advances to step 408, inwhich server A waits for one time interval. After waiting during thetime interval of step 408, server A attempts to detect reliablecommunication once again with server B in decision block 409. If serverA detects reliable communication with server B in step 409, the methodreturns to step 402. Repeatedly checking for reestablished communicationaccording to the loop defined by steps 407, 408, and 409 beforereforming the cluster prevents short duration non-fatal software errorsin server B from causing server A to begin operating in the place ofserver B.

[0052] If server A fails to detect reliable communication from server Bafter a required number of attempts, server A assigns itself the rightto survive in step 411 then reforms the cluster in step 412. Afterserver A reforms the cluster in step 412, it once again starts checkingfor reliable communication from server B at decision block 413. Ifserver B recovers from a non-terminating software error it might resumetransmitting a heartbeat signal and otherwise communicating with serverA after the cluster has been reformed. So long as communication fromserver B is not detected, server A continues operating and periodicallymonitors for communication from server B in the loop illustrated bydecision block 413 and step 414. If communication from server B isreestablished as detected at decision block 413, processing advances tostep 415, in which server B is shut down in step 415 before any diskaccess is permitted, thereby avoiding operation in a split brain modeand the associated data inconsistencies on the mass storage devices ofserver A and server B.

[0053] A primary server having right to survive performs the steps ofthe method of FIG. 5 to respond to a loss of communication from anotherserver according to one embodiment of the invention. In decision block501, the primary server B attempts to detect reliable communication withserver A. If server B can detect reliable communication with server A,the method proceeds to step 502, in which server B waits a certain timeinterval and then repeats decision block 501.

[0054] If server B does not detect reliable communication with server Ain decision block 501, the method proceeds to decision block 507, inwhich it is determined whether server B has waited the required numberof time intervals before it can assume that server A has failed. If,according to decision block 507, server B has not waited the requirednumber of time intervals, the method advances to step 508, in whichserver B waits for one time interval. After waiting during the timeinterval of step 508, server B attempts to detect reliable communicationonce again from server A in decision block 509. If server B detectsreliable communication with server A in step 509, the method returns tostep 502. Repeatedly checking for reestablished communication using theloop defined by steps 507, 508, and 509 before reforming the clusterprevents short duration non-fatal software errors in server A fromcausing server B to begin operating in the place of server A.

[0055] If server B fails to detect reliable communication from server Aafter a required number of attempts, server B reforms the cluster instep 512. After server B reforms the cluster in step 512, it once againstarts checking for reliable communication from server A at decisionblock 513. If server A recovers from a non-terminating software error itmight resume transmitting a heartbeat signal and otherwise communicatingwith server B after the cluster has been reformed. So long ascommunication from server A is not detected, server B continuesperiodically monitoring for communication from server A in the loopillustrated by decision block 513 and step 514. If communication fromserver A is reestablished as detected at decision block 513, processingadvances to step 516, in which server B reforms the cluster toaccommodate the resumed operation of server A.

[0056] The present invention may be embodied in other specific formswithout departing from its spirit or essential characteristics. Thedescribed embodiments are to be considered in all respects only asillustrative and not restrictive. The scope of the invention is,therefore, indicated by the appended claims rather than by the foregoingdescription. All changes that come within the meaning and range ofequivalency of the claims are to be embraced within their scope.

What is claimed and desired to be secured by United States LettersPatent is:
 1. A method for improving the availability and reliability ofa computer clustering system including a first server and a secondserver connected by a communication link, wherein said second server isassigned the right to survive in case of disruption in said computerclustering system, said method comprising the acts of: said first serverdetecting loss of communication from said second server to said firstserver; said first server analyzing the communication link to determineif the communication link is functioning properly; said first servercontinuing operation and assuming the right to survive if thecommunication link is determined to be functioning properly; and saidfirst server discontinuing operation if the communication link isdetermined to be not functioning properly.
 2. A method as recited inclaim 1 wherein the act of detecting loss of communication comprises theact of said first server determining that a heartbeat signal generatedby said second server cannot be detected on the communication link.
 3. Amethod as recited in claim 2, wherein the communication link comprises adedicated link that connects the first server and the second server. 4.A method as recited in claim 1 wherein the communication link includes avirtual channel established in a computer network, said computer networkcomprising one or more network devices interconnected to each other andinterconnected to said first server and said second server.
 5. A methodas recited in claim 1 wherein: each of said first server and said secondserver further comprises a file operating system and at least oneattached mass storage device; and each of said first server and saidsecond server executes said file operating system to become capable ofservicing network requests from other network devices, said networkrequests comprising requests to use the resources of said first serverand said second server.
 6. A method as recited in claim 5, furthercomprising the acts of: mirroring data from the at least one attachedstorage device of said first server to the at least one attached storagedevice of said second server; and mirroring data from the at least oneattached storage device of said second server to the at least oneattached storage device said first server.
 7. A method as recited inclaim 6 wherein the communication link is a dedicated communication linkthat interconnects only said first server and said second server, theact of detecting loss of communication comprising the acts of: saidfirst server attempting to communicate with said second server acrossthe communication link; and said first server analyzing the results ofthe attempted communication across said communication link to determinesaid first server cannot communicate with said second server.
 8. Amethod as recited in claim 1 wherein the act of analyzing thecommunication link to determine if said communication link isfunctioning properly further comprises the act of said first serverrepeatedly checking the transmission characteristics of thecommunication link to determine if they are within specified parameters.9. A method as recited in claim 1 wherein said first server determinessaid communication link is functioning properly, the method furthercomprising the act of said first server servicing network requests madeby a network device to said second server.
 10. A method as recited inclaim 9, further comprising the acts of: said first server determiningthat communication from said second server is restored; and said secondserver discontinuing operation.
 11. A method as recited in claim 1wherein said first server determines said communication link is notfunctioning properly, the method further comprising the act of saidsecond server continuing to operate and service network requests made tosaid first server after said first server discontinues operation.
 12. Amethod for improving the availability and reliability of a computerclustering system including a first server and a second serverinterconnected by a communication link, each of said first server andsaid second network server including a file server operating system andat least one associated mass storage device such that each of said firstserver and second server can receive requests that result in data beingwritten to or read from the associated at least one mass storage device,said second server being assigned right to survive in case of disruptionin said computer clustering system, said method comprising the acts of:said first server determining that a heartbeat signal associated withsaid second server is no longer being detected on said communicationlink; said first server analyzing the communication link to determine ifany error exists in the integrity of the communication link; if saidfirst server determines there is an error in the integrity of thecommunication link, said first server discontinuing operation and saidsecond server reforming the computer clustering system so that saidsecond server services requests that would otherwise be directed to saidfirst server; and if said first server determines that there is no errorin the integrity of the communication link, said first server assigningitself the right to survive and said first server reconfiguring thecomputer clustering system so that said first server services requeststhat would otherwise be directed to said second server.
 13. A method asrecited in claim 12, further comprising, prior to the act of determiningthat the heartbeat signal associated with the second server is no longerbeing detected, the act of said first server and said second servermirroring data stored on the at least one mass storage device of thefirst server and the at least one mass storage device of the secondserver.
 14. A method as recited in claim 13, wherein the act ofmirroring data comprises transmitting the data on the communication linkbetween the first server and the second server.
 15. A method as recitedin claim 13, wherein the act of said first server determining that theheartbeat signal associated with said second server is no longer beingdetected comprises the acts of: said first server repeatedly monitoringthe communication link for the heartbeat signal; and said first serverdetermining that the heartbeat signal is no longer being detected whensaid first server does not detect the heartbeat signal during the act ofrepeatedly monitoring the communication link during a specified periodof time.
 16. A method as recited in claim 12, further comprising, afterthe act of said first server reconfiguring the computer clusteringsystem, the acts of: said first server again detecting the heartbeatsignal associated with the second server; and said second serverdiscontinuing operation prior to accessing the at least one mass storagedevice associated with said second server.
 17. A method for improvingthe availability and reliability of a computer clustering systemincluding a first server and a second server interconnected by acommunication link, each of said first server and said second networkserver including a file server operating system and at least oneassociated mass storage device such that each of said first server andsecond server can receive requests that result in data being written toor read from the associated at least one mass storage device, saidsecond server being assigned right to survive in case of disruption insaid computer clustering system, said method comprising the acts of:during normal operation of the computer clustering system, mirroringdata on the at least one mass storage device associated with the firstserver and on the at least one mass storage device associated with thesecond server by transmitting the data between the first server and thesecond server using the communication link; said first serverdetermining that said second server is not functioning properly,including the acts of: said first server analyzing the communicationlink at specified time intervals; said first server failing to detectcommunication from said second server on said communication link; andsaid first server detecting proper functionality of said communicationlink based on the act of analyzing said communication link; said firstserver taking control of the computer clustering system; and said firstserver reconfiguring the computer clustering system so said first serverreceives file server requests that would otherwise be directed to saidsecond server.
 18. A method as recited in claim 17, further comprisingthe acts of: said first server determining that said second server hasreestablished proper functionality after said first server has takencontrol of the computer clustering system; and said second serverdiscontinuing operation prior to accessing said at least one massstorage device associated with said second server.
 19. A computerprogram product for implementing, in a first server included in acomputer clustering system that also includes a second server and acommunication link connecting the first server and the second server, amethod for said first server to assume control of the computerclustering system in response to a failure of said second server, thecomputer program product comprising: a computer-readable medium carryingcomputer-executable instructions for implementing the method, saidcomputer-executable instructions including: program code means fordetermining whether said first server has a right to survive if saidfirst server fails to detect communication from said second server, saidfirst server initially not having the right to survive; program codemeans for detecting loss of communication from said second server;program code means for determining, in response to the loss ofcommunication, whether the communication link is functioning properly;program code means for executing the acts of: if the communication linkis not functioning properly, discontinuing operation of said firstserver; and if the communication link is functioning properly,continuing operation of said first server, notwithstanding said firstserver initially not having the right to survive.
 20. A computer programproduct as recited in claim 19, wherein the communication link comprisesa dedicated link between the first server and the second server.
 21. Acomputer program product as recited in claim 19, wherein thecommunication link comprises a virtual channel included in a networkthat interconnects the first server, the second server, and a pluralityof network devices.
 22. A computer program product as recited in claim19, wherein the computer-executable instructions further compriseprogram code means for mirroring data between a mass storage deviceassociated with the first server and a mass storage device associatedwith the second server during normal operation of the computerclustering system, while both the first server and the second servercommunicate one with another.
 23. A computer program product as recitedin claim 19, wherein the program code means for determining whether saidfirst server has a right to survive comprises program code means forexamining a right to survive flag stored at the first server.